Independent Component Analysis Using Jaynes’ Maximum Entropy Principle

ICA deals with finding linear projections in the input space along which the data shows most independence. Therefore, mutual information between the projected outputs, which are usually called the separated outputs due to links with blind source separation (BSS), is considered to be a natural criterion for ICA. Minimization of the mutual information requires primarily the estimation of this quantity from the samples, and then adaptation of the separation matrix parameters using a suitable optimization approach. In this paper, we present a numerical procedure to estimate an upper bound for the mutual information based on density estimates motivated by Jaynes’ maximum entropy principle. The gradient of the mutual information with respect to the adaptive parameters, then turns out to be extremely simple.